Systems and methods for generating views of unmanned aerial vehicles

ABSTRACT

A method for generating a view for an unmanned aerial vehicle is provided. The method includes obtaining an origin and a destination of the unmanned aerial vehicle, determining a group of imaging devices based on a route between the origin and the destination of the unmanned aerial vehicle, and obtaining a view of the unmanned aerial vehicle following the route based on images of the unmanned aerial vehicle captured by the group of imaging devices. The group of imaging devices form one or more collaborative camera sensor networks. The collaborative camera sensor networks identify an unmanned aerial vehicle and create human operator&#39;s view of the unmanned aerial vehicle. The route of the unmanned aerial vehicle is determined based on the availability of imaging devices of the camera sensor network. The route of the unmanned aerial vehicle may be changed based on the availability of imaging devices.

TECHNICAL FIELD

The present specification generally relates to systems and methods forgenerating a view of an unmanned aerial vehicle and, more specifically,to systems and methods for generating a view of an unmanned aerialvehicle based on available imaging devices that are selected based onthe route of the unmanned aerial vehicle.

BACKGROUND

Unmanned aerial vehicles are used in various industries includingagriculture, security and surveillance, delivery of goods and services,and telecommunication. However, Federal Aviation Administrationregulations (e.g., restricted zones, line-of-sight of human operatorrestrictions, see-and-avoid requirements, etc.) limit the functionalityof unmanned aerial vehicles. Particularly, when an unmanned aerialvehicle flies a long-distance, the unmanned aerial vehicle may besubject to extensive restricted zones and be out of line-of-sight of ahuman operator, which may fail to meet the see-and-avoid requirement.

Accordingly, a need exists for obtaining a real-time view of an unmannedaerial vehicle traveling a long distance.

SUMMARY

In one embodiment, a method for generating a view for an unmanned aerialvehicle is provided. The method includes obtaining an origin and adestination of the unmanned aerial vehicle, determining a group ofimaging devices based on a route between the origin and the destinationof the unmanned aerial vehicle, and obtaining a view of the unmannedaerial vehicle following the route based on images of the unmannedaerial vehicle captured by the group of imaging devices.

In another embodiment, a system for generating a view for an unmannedaerial vehicle is provided. The system includes an electronic controlunit configured to: obtain an origin and a destination of the unmannedaerial vehicle, determine a group of imaging devices based on a routebetween the origin and the destination of the unmanned aerial vehicle,receive images captured by the group of imaging devices, and obtain aview of the unmanned aerial vehicle following the route based on thereceived images.

In yet another embodiment, a method for generating a view for anunmanned aerial vehicle is provided. The method includes determining aroute for the unmanned aerial vehicle based on a current location of theunmanned aerial vehicle, selecting a group of imaging devices based onthe route, and obtaining a view of the unmanned aerial vehicle followingthe route based on images captured by the group of imaging devices.

These and additional features provided by the embodiments of the presentdisclosure will be more fully understood in view of the followingdetailed description, in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the disclosure. The followingdetailed description of the illustrative embodiments can be understoodwhen read in conjunction with the following drawings, where likestructure is indicated with like reference numerals and in which:

FIG. 1 depicts, a system for generating a view of an unmanned aerialvehicle using camera sensor networks, according to one or moreembodiments shown and described herein;

FIG. 2 depicts schematic diagrams of the system for generating a view ofan unmanned aerial vehicle using camera sensor networks, according toone or more embodiments shown and described herein;

FIG. 3 depicts a handover between camera sensor networks, according toone or more embodiments shown and described herein;

FIG. 4 depicts an exemplary scenario where an unmanned aerial vehiclemoves out of camera sensor networks, according to one or moreembodiments shown and described herein;

FIG. 5 is a flowchart for obtaining a view of an unmanned aerial vehicleusing camera sensor networks, according to one or more embodiments shownand described herein;

FIG. 6 depicts exemplary camera sensor networks, according to one ormore embodiments shown and described herein;

FIG. 7 is a flowchart for obtaining a view of an unmanned aerial vehicleusing camera sensor networks, according to another embodiment shown anddescribed herein;

FIG. 8 depicts exemplary camera sensor networks, according to one ormore embodiments shown and described herein;

FIG. 9 depicts a flowchart for obtaining a view of an unmanned aerialvehicle using camera sensor networks when the origin and destination ofthe unmanned aerial vehicle is not known to a system, according toanother embodiment shown and described herein;

FIG. 10A depicts exemplary camera sensor networks, according to one ormore embodiments shown and described herein; and

FIG. 10B depicts exemplary camera sensor networks, according to one ormore embodiments shown and described herein.

DETAILED DESCRIPTION

The embodiments disclosed herein include systems and methods forobtaining a real time view of an unmanned aerial vehicle. Referringgenerally to FIGS. 1 and 5, a method for generating a view for anunmanned aerial vehicle is provided. The method includes obtaining anorigin 162 and a destination 164 of the unmanned aerial vehicle 104,determining a group of imaging devices based on a route between theorigin 162 and the destination 164 of the unmanned aerial vehicle 104,and obtaining a view of the unmanned aerial vehicle following the routebased on images of the unmanned aerial vehicle captured by the group ofimaging devices.

The group of imaging devices form one or more collaborative camerasensor networks. The collaborative camera sensor networks identify anunmanned aerial vehicle and create a human operator's view of theunmanned aerial vehicle. The route of the unmanned aerial vehicle isdetermined based on the availability of imaging devices of the camerasensor network. The route of the unmanned aerial vehicle may be changedbased on the availability of imaging devices. Whenever the unmannedaerial vehicle moves out of a camera sensor network region, horizontaland/or vertical hand over is performed between camera sensor networks asshown in FIG. 3 to continuously monitor the flight of the unmannedaerial vehicle. If the unmanned aerial vehicle is at a location notcovered by a camera sensor network, a special mission drone may bepositioned proximate to the unmanned aerial vehicle as shown in FIG. 4.The image data obtained by the camera sensor networks is transferred toan edge server and/or a cloud server, which in turn transmits thecaptured images to a human operator.

FIG. 1 depicts, a system for generating a view of an unmanned aerialvehicle using camera sensor networks, according to one or moreembodiments shown and described herein. In FIG. 1, a system 100 mayinclude a cloud server 102, an unmanned aerial vehicle 104, edge servers140 and 150, and camera sensor networks 110 and 120.

A user 160 of the unmanned aerial vehicle 104 is located remotely fromthe unmanned aerial vehicle 104. The user 160 may remotely control theunmanned aerial vehicle 104 based on a view of the unmanned aerialvehicle received from camera sensor networks, e.g., the camera sensornetwork 110.

In embodiments, the unmanned aerial vehicle 104 may share its origin anddestination information with an edge server and/or a cloud server. Forexample, the unmanned aerial vehicle 104 may transmit its origin 162 anddestination 164 to a nearby edge server, e.g., the edge server 140, orto the cloud server 102. The edge server 140 and/or cloud server 102 maydynamically form one or more collaborative camera sensor networks basedon a route between the origin 162 and destination 164 of the unmannedaerial vehicle 104 and available imaging devices. For example, in FIG.1, first and second collaborative camera sensor networks 110 and 120 areformed. The first collaborative camera sensor network 110 includes aplurality of imaging devices. For example, the first collaborativecamera sensor network 110 includes imaging devices 112 and 114 of movingvehicles, and road-side imaging devices 116, 118, and 119. The secondcollaborative camera sensor network 120 includes imaging devices 122,124, 126, and 128 of moving vehicles, and road-side imaging devices 132,134, 136, 138, and 139.

As the unmanned aerial vehicle 104 travels from the origin 162 to thedestination 164 following the route 166, a view of the unmanned aerialvehicle 104 is obtained based on images of the unmanned aerial vehicle104 captured by a plurality of imaging devices of collaborative camerasensor networks. For example, the imaging device 114 may initiatecapturing of the unmanned aerial vehicle 104 as the unmanned aerialvehicle 104 travels following the route 166. The imaging device 114 mayconstantly transmit captured images to the edge server 140 or the cloudserver 102 which in turn transmits the captured images to the user 160(e.g., a device 163 of the user 160). When the unmanned aerial vehicle104 comes proximate to the road-side imaging device 116 (e.g., when theunmanned aerial vehicle 104 is within a viewing distance of theroad-side imaging device 116), the road-side imaging device 116 maycapture images of the unmanned aerial vehicle 104 and transmit theimages to the edge server 140 or the cloud server 102 which in turntransmits the images to the user 160.

In embodiments, more than one imaging device may capture images of theunmanned aerial vehicle 104 at the same time. For example, both theimaging device 114 and the imaging device 116 may capture images of theunmanned aerial vehicle 104 at the same time from differentperspectives. Both the imaging devices 114 and 116 may transmit capturedimages to the edge server 140. The edge server 140 may synthesize theimages to obtain an enhanced view of the unmanned aerial vehicle 104(e.g., a 360 degree view of the unmanned aerial vehicle 104).Alternatively, the edge server 140 transmits the captures images to thecloud server 102, and the cloud server 102 may synthesize the images toobtain an enhanced view of the unmanned aerial vehicle 104.

In embodiments, the edge servers 140 or 150 or the cloud server 102processes the images captured by the imaging devices and determineswhether the unmanned aerial vehicle 104 shows any unusual actions. If itis determined that the unmanned aerial vehicle 104 shows any unusualactions, the edge server 140 or 150 or the cloud server 102 may transmitan alert to the user 160. In some embodiments, the edge server 140 or150 or the cloud server 102 may determine the current location of theunmanned aerial vehicle 104 based on captured images and determinewhether the unmanned aerial vehicle 104 is within in a restricted zonebased on the location and pre-stored information about restricted zones.If it is determined that the unmanned aerial vehicle 104 is within arestricted zone, the edge server 140 or 150 or the cloud server 102 maytransmit an alert to the user 160.

FIG. 2 depicts schematic diagrams of the system for generating a view ofan unmanned aerial vehicle using camera sensor networks, according toone or more embodiments shown and described herein.

The unmanned aerial vehicle 104 includes one or more processors 202, oneor more memory modules 204, a satellite antenna 206, a network interfacehardware 208, one or more cameras 210, and a beacon device 212.

Each of the one or more processors 202 of the unmanned aerial vehicle104 may be any device capable of executing machine readableinstructions. Accordingly, each of the one or more processors 202 may bea controller, an integrated circuit, a microchip, a computer, or anyother computing device. Each of the one or more processors 202 iscommunicatively coupled to the other components of the unmanned aerialvehicle 104 by the communication path 214. Accordingly, thecommunication path 214 may communicatively couple any number ofprocessors with one another, and allow the components coupled to thecommunication path 214 to operate in a distributed computingenvironment. Specifically, each of the components may operate as a nodethat may send and/or receive data.

Each of the one or more memory modules 204 of the unmanned aerialvehicle 104 is coupled to the communication path 214 and communicativelycoupled to the one or more processors 202. Each of the one or morememory modules 204 may comprise RAM, ROM, flash memories, hard drives,or any device capable of storing machine readable instructions such thatthe machine readable instructions can be accessed and executed by theone or more processors 202. The machine readable instructions maycomprise logic or algorithm(s) written in any programming language ofany generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, for example,machine language that may be directly executed by the one or moreprocessors 202, or assembly language, object-oriented programming (OOP),scripting languages, microcode, etc., that may be compiled or assembledinto machine readable instructions and stored in the one or more memorymodules 204. Alternatively, the machine readable instructions may bewritten in a hardware description language (HDL), such as logicimplemented via either a field-programmable gate array (FPGA)configuration or an application-specific integrated circuit (ASIC), ortheir equivalents. Accordingly, the functionality described herein maybe implemented in any conventional computer programming language, aspre-programmed hardware elements, or as a combination of hardware andsoftware components.

The one or more memory modules 204 may include the origin anddestination of the unmanned aerial vehicle 104, and an assigned routebetween the origin and the destination received from the edge server 140or the cloud server 102. The one or more processors 202 may operate oneor more electric motors of the unmanned aerial vehicle 104 to followsthe assigned route.

Still referring to FIG. 2, a satellite antenna 206 is coupled to thecommunication path 214 such that the communication path 214communicatively couples the satellite antenna 206 to other modules ofthe unmanned aerial vehicle 104. The satellite antenna 206 is configuredto receive signals from global positioning system satellites.Specifically, in one embodiment, the satellite antenna 206 includes oneor more conductive elements that interact with electromagnetic signalstransmitted by global positioning system satellites. The received signalis transformed into a data signal indicative of the location (e.g.,latitude, longitude, and altitude) of the satellite antenna 206 or anobject positioned near the satellite antenna 206, by the one or moreprocessors 202. The one or more memory modules 204 may includeinstructions for transmitting the location received by the satelliteantenna 206 to the edge device 140 or the cloud server 102.

Still referring to FIG. 2, the network interface hardware 208 is coupledto the communication path 214 and communicatively coupled to the one ormore processors 202. The network interface hardware 208 may be anydevice capable of transmitting and/or receiving data via a network.Accordingly, the network interface hardware 208 can include acommunication transceiver for sending and/or receiving any wired orwireless communication. For example, the network interface hardware 208may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card,mobile communications hardware, near-field communication hardware,satellite communication hardware and/or any wired or wireless hardwarefor communicating with other networks and/or devices. In someembodiments, the network interface hardware 208 includes hardwareconfigured to operate in accordance with the Bluetooth wirelesscommunication protocol. In other embodiments, the network interfacehardware 208 includes hardware configured to operate in accordance witha wireless communication protocol other than Bluetooth. The networkinterface hardware 208 of the unmanned aerial vehicle 104 maycommunicate with the edge server 140 or the cloud server 102.

Still referring to FIG. 2, one or more cameras 210 are coupled to thecommunication path 214 such that the communication path 214communicatively couples the one or more cameras 210 to other modules ofthe unmanned aerial vehicle 104. Each of the one or more cameras 210 maybe any device having an array of sensing devices (e.g., pixels) capableof detecting radiation in an ultraviolet wavelength band, a visiblelight wavelength band, or an infrared wavelength band. Each of the oneor more cameras 210 may have any resolution. The one or more cameras 210may include an omni-directional camera, or a panoramic camera. In someembodiments, one or more optical components, such as a mirror, fish-eyelens, or any other type of lens may be optically coupled to at least oneof the one or more cameras 210. The one or more cameras 210 may be usedto capture an image of another unmanned aerial vehicle.

Still referring to FIG. 2, the communication path 214 may be formed fromany medium that is capable of transmitting a signal such as, forexample, conductive wires, conductive traces, optical waveguides, or thelike. Moreover, the communication path 214 may be formed from acombination of mediums capable of transmitting signals. In oneembodiment, the communication path 214 comprises a combination ofconductive traces, conductive wires, connectors, and buses thatcooperate to permit the transmission of electrical data signals tocomponents such as processors, memories, sensors, input devices, outputdevices, and communication devices. Accordingly, the communication path214 may comprise a bus. Additionally, it is noted that the term “signal”means a waveform (e.g., electrical, optical, magnetic, mechanical orelectromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave,square-wave, vibration, and the like, capable of traveling through amedium. The communication path 214 communicatively couples the variouscomponents of the unmanned aerial vehicle 104. As used herein, the term“communicatively coupled” means that coupled components are capable ofexchanging data signals with one another such as, for example,electrical signals via conductive medium, electromagnetic signals viaair, optical signals via optical waveguides, and the like.

Still referring to FIG. 2, a beacon device 212 is coupled to thecommunication path 214 and communicatively coupled to the one or moreprocessors 202. The beacon device 212 may transmits a wireless beaconsignal to devices nearby. The beacon signal may include identificationinformation about the unmanned aerial vehicle 104. For example, theimaging device 114 shown in FIG. 1 receives the wireless beacon signalfrom the unmanned aerial vehicle 104 and identifies the unmanned aerialvehicle 104.

Now referring to the cloud server 102, the one or more processors 220may be processors similar to the one or more processors 202 describedabove. The one or more memory modules 230 may be memories similar to theone or more memory modules 204 described above. The network interfacehardware 240 may be an interface hardware similar to the networkinterface hardware 208 described above. The communication path 250 maybe a communication path similar to the communication path 214 describedabove. The one or more processors 220 in combination of one or morememory modules 230 may operate as an electronic control unit for thecloud server 102.

The one or more memory modules 230 includes a camera database 232, anunmanned aerial vehicle information database 234, a route planner module236, and a remote person view module 238. The camera database 232 storeslocations and specifications of imaging devices. The imaging devices mayinclude mobile imaging devices, e.g., imaging devices of mobile phones,imaging devices of unmanned aerial vehicles, imaging devices ofvehicles, and the like. In addition, the imaging devices may includeimaging devices at fixed locations, e.g., imaging devices at trafficlights, imaging devices of security cameras, imaging devices of roadsideunits. For mobile imaging devices, the camera database 232 may includepredicted locations of the mobile imaging devices at a particular time.For the imaging devices at fixed locations, the locations of the imagingdevices are stored in the camera database 232. The specification of theimaging devices include angle of view, resolution, and a viewingdistance of each imaging devices.

The unmanned aerial vehicle information database 234 may includeidentifications of unmanned aerial vehicles, and origin and destinationinformation for each unmanned aerial vehicle. The origin and destinationof each unmanned aerial vehicle may be received from each of theunmanned aerial vehicles. As another example, a user of the unmannedaerial vehicle may transmit the origin and destination of the unmannedaerial vehicle to the cloud server 102 by inputting the origin anddestination on a controller for the unmanned aerial vehicle.

The route planner module 236 may determine a route for each unmannedaerial vehicle based on information stored in the camera database 232and the unmanned aerial vehicle information database 234. For example,for the unmanned aerial vehicle 104, based on the origin and thedestination of the unmanned aerial vehicle 104 and locations of imagingdevices between the origin and destination stored in the camera database232, a route is planned such that the unmanned aerial vehicle 104following the route is constantly captured by at least one of theimaging devices.

The remote person view module 238 provides a view of an unmanned aerialvehicle to the user 160 of the unmanned aerial vehicle or the wearabledevice 163 of the user 160. In embodiments, the remote person viewmodule 238 may receive captured images of an unmanned aerial vehicle 104from imaging devices and transmit the captured images to the user 160 orthe wearable device 163 of the user. In some embodiments, the remoteperson view module 238 may synthesize captured images including theunmanned aerial vehicle 104 and transmit the synthesized images to theuser 160. In some embodiments, the remote person view module 238 mayinstruct each imaging devices to directly communicate with the user 160or the wearable device 163 of the user 160. For example, the remoteperson view module 238 may instruct the imaging device 114 to transmitcaptured images of the unmanned aerial vehicle 104 to the user 160. Asthe unmanned aerial vehicle 104 moves away from the imaging device 114and approaches the imaging device 116, the remote person view module 238may instruct the imaging device 116 to transmit captured images of theunmanned aerial vehicle 104 to the user 160. In some embodiments, boththe imaging device 114 and the imaging device 116 may transmit thecaptured images to the edge server 140, and the edge server 140 maycombine or stitch the captured images to obtain an enhanced view of theunmanned aerial vehicle 104. Then, the edge server 140 may forward thecombined or stitched images to the user 160.

Each of the route planner module 236 and the remote person view module238 may be a program module in the form of operating systems,application program modules, and other program modules stored in one ormore memory modules 230. In some embodiments, the program module may bestored in a remote storage device that may communicate with the cloudserver 102. Such a program module may include, but is not limited to,routines, subroutines, programs, objects, components, data structures,and the like for performing specific tasks or executing specific datatypes as will be described below.

The edge server 140 may include similar components as the cloud server102. For example, the edge server 140 may include one or moreprocessors, one or more memory modules, and a network hardwareinterface. The one or more memory modules may include a camera database,an unmanned aerial vehicle information database, a route planner module,and a remote person view module, similar to the one or more memorymodules 230 of the cloud server 102.

FIG. 3 depicts a handover between camera sensor networks, according toone or more embodiments shown and described herein. In FIG. 3, at timet=to, the unmanned aerial vehicle 104 is in an area covered by the firstcollaborative camera sensor network 110. The first collaborative camerasensor network 110 includes a plurality of imaging devices. For example,the first collaborative camera sensor network 110 includes imagingdevices 112 and 114 of moving vehicles, and road-side imaging devices116, 118, and 119. The second collaborative camera sensor network 120includes imaging devices 122, 124, 126, and 128 of moving vehicles, androad-side imaging devices 132, 134, 136, 138, and 139.

When the unmanned aerial vehicle 104 is within the area covered by thecamera sensor network 110 , the imaging device 114 may initiatecapturing of the unmanned aerial vehicle 104 as the unmanned aerialvehicle 104 travels following the route 166. The imaging device 114 mayconstantly transmit captured images to the edge server 140 which in turntransmits the images to the cloud server 102 or to the user 160 (e.g., awearable device 163 of the user 160). In some embodiments, the imagingdevice 114 may transmit captured images to the cloud server 102 which inturn transmits the captured images to the user 160.

When the unmanned aerial vehicle 104 moves out of the area covered bythe camera sensor network 110, i.e., none of the imaging devices of thecollaborative camera sensor network 120 is capable of capturing theimage of the unmanned aerial vehicle 104, hand overring to anothercamera sensor network may be performed. For example, at time t=t₁, theunmanned aerial vehicle 104 moves out of the area covered by the camerasensor network 110. The unmanned aerial vehicle 104 is now within anarea covered by the camera sensor network 120, and one of the imagingdevices, e.g., the imaging device 124, may capture images of theunmanned aerial vehicle 104 and transmit the images to the edge server150 which in turn transmits the images to the cloud server 102 or to theuser 160. In this regard, a handover between camera sensor networks 110and 120 is implemented as the unmanned aerial vehicle 104 travels alongthe route 166.

FIG. 4 depicts an exemplary scenario where an unmanned aerial vehiclemoves out of camera sensor networks, according to one or moreembodiments shown and described herein. In FIG. 4, the unmanned aerialvehicle 104 moves from a position P₁ to a position P₂. When the unmannedaerial vehicle 104 is at the position P₁, one of the imaging devices,for example, the imaging device 114, captures images of the unmannedaerial vehicle 104 and transmits the captured images to the user 160 viathe edge server 140 and/or the cloud server 102. When the unmannedaerial vehicle 104 moves to the position P₂, the unmanned aerial vehicle104 is out of areas covered by the camera sensor networks 110 and 120.Thus, none of the imaging devices of the camera sensor network 110 and120 is able to capture images of the unmanned aerial vehicle 104. Theedge server 140 or 150 may transmit a notification to the cloud server102 that the unmanned aerial vehicle 104 is out of areas covered byavailable camera sensor networks, e.g., camera sensor networks 110 and120. The cloud server 102 may then identify the current location of theunmanned aerial vehicle 104 and instruct an unmanned aerial vehicle 410to move towards the current location of the unmanned aerial vehicle 104.The unmanned aerial vehicle 410 includes an imaging device 412 thatcaptures images of the unmanned aerial vehicle 104. The unmanned aerialvehicle 410 may transmit captured images to the user 160 via an edgeserver proximate to the unmanned aerial vehicle 410 and/or the cloudserver 102.

In embodiments, the unmanned aerial vehicle 410 follows the unmannedaerial vehicle 104 and continuously captures the images of the unmannedaerial vehicle 410. In some embodiments, the unmanned aerial vehicle 410may lead the unmanned aerial vehicle 104 to move back to the areacovered by the camera sensor network 110 or 120. For example, theunmanned aerial vehicle 410 may provide a direction to the area coveredby the camera sensor network 110 or 120 to the unmanned aerial vehicle104. In some embodiments, the cloud server 102 may provide a directionto the area covered by the camera sensor network 110 or 120 to theunmanned aerial vehicle 104.

FIG. 5 is a flowchart for obtaining a view of an unmanned aerial vehicleusing camera sensor networks, according to one or more embodiments shownand described herein.

In step 510, a system identifies an unmanned aerial vehicle. Inembodiments, the unmanned aerial vehicle may be identified by an edgeserver based on wireless beaconing submitted from the unmanned aerialvehicle. For example, by referring to FIG. 1, the edge server 140 or oneof the imaging devices in the camera sensor network 110 may receive awireless beaconing signal from the unmanned aerial vehicle 104 andidentify the unmanned aerial vehicle 104. In another embodiment, anunmanned aerial vehicle may be identified based on image-based depthestimation by imaging devices in the camera sensor network 110. Forexample, by referring to FIG. 1, one of the imaging devices in thecamera sensor network 110 may identify the unmanned aerial vehicle 104using image-based depth estimation.

In step 520, the system obtains an origin and a destination of theidentified unmanned aerial vehicle. In embodiments, the unmanned aerialvehicle may communicate its origin and destination to an edge server ora cloud server. For example, by referring to FIG. 1, the unmanned aerialvehicle 104 may communicate information about the origin 162 and thedestination 164 to the edge server 140 or the cloud server 102 alongwith the identification of the unmanned aerial vehicle 104. In someembodiments, the origin and destination of the unmanned aerial vehiclein association with the identification of the unmanned aerial vehiclemay be pre-stored in the cloud server 102. For example, the cloud server102 may receive and store the origin and destination of the unmannedaerial vehicle from a user of the unmanned aerial vehicle along with theidentification of the unmanned aerial vehicle.

In step 530, the system determines a group of imaging devices based on aroute between the origin and the destination of the unmanned aerialvehicle. In embodiments, the cloud server 102 may determine a route forthe unmanned aerial vehicle based on the origin and the destination ofthe unmanned aerial vehicle. For example, by referring to FIG. 6, thecloud server 102 may receive the origin 162 and the destination 164 ofthe unmanned aerial vehicle 104. The cloud server 102 may determine aroute including a path 602, a path 604, and a path 606. In embodiments,the cloud server 102 may determine imaging devices that are availablefor capturing images of the unmanned aerial vehicle 104 following theroute.

The cloud server 102 or a corresponding edge server may determinewhether imaging devices are available for capturing images capturingimages of the unmanned aerial vehicle based on various factors, andselect the available imaging devices as the group of imaging devices.For example, the cloud server 102 or a corresponding edge server maydetermine whether imaging devices are available for capturing images ofthe unmanned aerial vehicle based on the proximity of the imagingdevices to the route of the unmanned aerial vehicle.

As another example, the cloud server 102 or a corresponding edge servermay determine whether imaging devices are available for capturing imagesof the unmanned aerial vehicle based on expected locations of mobileimaging devices. The mobile imaging devices may include, but is notlimited to, an imaging device of a mobile phone, an imaging device ofanother unmanned aerial vehicle, or an imaging device of a vehicle. Thecloud server 102 or a corresponding edge server may determine theexpected locations of the mobile imaging devices when the unmannedaerial vehicle 104 follows the route including the paths 602, 604, 606and determine whether the expected locations of the mobile imagingdevices are proximate to the route. Specifically, by referring to FIG.6, the cloud server 102 may determine the expected location of theimaging device of another unmanned aerial vehicle 650 when the unmannedaerial vehicle 104 follows the path 602. If the expected location 652 ofanother unmanned aerial vehicle is proximate to the path 602 when theunmanned aerial vehicle 104 follows the path 602, then the cloud server102 may determine that the imaging device of another unmanned aerialvehicle 650 is available for capturing images of the unmanned aerialvehicle 104.

As another example, the cloud server 102 or a corresponding edge servermay determine whether imaging devices are available for capturing imagesof the unmanned aerial vehicle based on face directions of the imagingdevices. The face directions of imaging devices may be stored in thecloud server, e.g., in the camera database 232 in FIG. 2 or in acorresponding edge server. The cloud server 102 or a corresponding edgeserver may determine whether an imaging device is facing the unmannedaerial vehicle 104 following the route based on the face direction ofthe imagine device and the moving direction of the unmanned aerialvehicle 104 following the route. If it is determined that the imagingdevice is facing the unmanned aerial vehicle 104 when the unmannedaerial vehicle 104 follows the route, the cloud server 102 may determinethat the imaging device is available for capturing images of theunmanned aerial vehicle 104. Specifically, by referring to FIG. 6, theimaging device 610-1 is facing the unmanned aerial vehicle 104 when theunmanned aerial vehicle 104 follows the path 602. Thus, the cloud server102 or a corresponding edge server may determine that the imaging device610-1 is available for capturing images of the unmanned aerial vehicle104. In contrast, the imaging device 610-2 is not facing the unmannedaerial vehicle 104 when the unmanned aerial vehicle 104 follows the path602. Thus, the cloud server 102 or a corresponding edge server maydetermine that the imaging device 610-2 is not available for capturingimages of the unmanned aerial vehicle 104.

As another example, the cloud server 102 may determine whether imagingdevices are available for capturing images of the unmanned aerialvehicle based on whether the imaging devices are registered into thesystem. For example, if an imaging device 620-1 has opted in toparticipate a camera sensor network, e.g., a camera sensor network 620in FIG. 6, the cloud server 102 may determine that the imaging device620-1 is available for capturing images of the unmanned aerial vehicle104. As another example, if an imaging device 620-2 has opted out fromparticipating the camera sensor network, then the cloud server 102 maydetermine that the imaging device 620-2 is not available for capturingimages of the unmanned aerial vehicle 104.

In embodiments, one or more camera sensor networks are formed based onthe imaging devices available for capturing images of the unmannedaerial vehicle. For example, by referring to FIG. 6, a camera sensornetwork 610, a camera sensor network 620, and a camera sensor network630 are formed. Each of the camera sensor networks 610, 620, and 630includes one or more imaging devices available for capturing images ofthe unmanned aerial vehicle 104. While FIG. 6 depict three differentcamera sensor networks, more than or less than three camera sensornetworks may be formed depending on distances among the imaging devices.

In embodiments, the system may provide incentives to the image devicesavailable for capturing images of the unmanned aerial vehicles such thatmore imaging devices are opted in to participate in a camera sensornetwork. The incentives may be determined based on at least one of adistance between the route and each imaging device, a view angle of eachimaging device, or a viewing distance of each imaging device.

Referring back to FIG. 6, in step 540, the system determines whether theunmanned aerial vehicle is within a view of at least one of the group ofimaging devices. By referring to FIG. 6, the cloud server 102 or an edgeserver for one of the camera sensor networks 610, 620, 630 may determinewhether the unmanned aerial vehicle 104 is within a view of at least oneof the group of imaging devices. For example, the cloud server 102 mayreceive the current location of the unmanned aerial vehicle 104 anddetermine whether the current location of the unmanned aerial vehicle104 is within view of the imaging devices of the camera sensor networks610, 620, and 630.

If it is determined that the unmanned aerial vehicle 104 is within aview of at least one of the group of imaging devices in step 540, thesystem obtains a view of the unmanned aerial vehicle following the routebased on images of the unmanned aerial vehicle captured by the group ofimaging devices in step 550. For example, by referring to FIG. 6, whenthe unmanned aerial vehicle 104 is at the origin 162, the imaging device610-1 may initiate capturing images of the unmanned aerial vehicle 104.The imaging device 610-1 may transmit the captured images to an edgeserver for the camera sensor network 610 or the cloud server 102 shownin FIG. 1 which in turn transmits the captured images to the device 163of the user 160. As the unmanned aerial vehicle 104 passes the imagingdevice 610-1, the imaging device 610-3 may capture images of theunmanned aerial vehicle 104 and transmit the captured images to the edgeserver or the cloud server. As the unmanned aerial vehicle 104 followsthe path 604, the imaging device 620-1 may capture images of theunmanned aerial vehicle 104 and transmit the captured images to the edgeserver or the cloud server. At this time, a hand over is implementedbetween the camera sensor network 610 and the camera sensor network 620.In this regard, at least one imaging device in the camera sensornetworks 610, 620, 630 captures images of the unmanned aerial vehicle104 in real time and transmits the captured images to the user 160 suchthat the user 160 can see the real time view of the unmanned aerialvehicle 104.

In some embodiments, the imaging devices of the camera sensor networksmay store captured images of the unmanned aerial vehicle for apredetermined time. The imaging devices may pre-process of capturedimages and store images that include the unmanned aerial vehicle for apredetermined time.

In some embodiments, the edge server or the cloud server may determine amisbehaving imaging device among the group of the imaging devices. Theedge server or the cloud server may compare images of the unmannedaerial vehicle captured by the group of imaging devices. The edge serveror the cloud server may identify a misbehaving imaging device of thegroup of imaging devices that provides images not congruent with imagesprovided by other imaging devices of the group of imaging devices. Theedge server or cloud server may compare the images captured by theimaging device 610-1, 620-1, 620-2 and determine that the image providedby the imaging device 620-2 is not congruent with the imagines providedby the imaging devices 610-1 and 620-1. For example, the imaging device620-2 may transmit an image of the unmanned aerial vehicle 104 thelocation of which does not match with the location of the unmannedaerial vehicle 104 in the images captured by the imaging devices 610-1and 620-1. As another example, the imaging device 620-2 may transmitimages with no unmanned aerial vehicle while the imaging devices 610-1and 620-1 transmit images including the unmanned aerial vehicle 104.Then, the edge server or the cloud server may discard the imagesprovided by the misbehaving imaging device, e.g., the imaging device620-2.

If it is determined that the unmanned aerial vehicle 104 is not within aview of at least one of the group of imaging devices in step 540, thesystem dispatches another unmanned aerial vehicle proximate to theunmanned aerial vehicle in response to the unmanned aerial vehicle beingoutside the view of the group of imaging devices in step 560. Forexample, when the unmanned aerial vehicle 104 moves to the position P3in FIG. 6, the unmanned aerial vehicle 104 is out of areas covered bythe camera sensor networks 610, 620, and 630. Thus, none of the imagingdevices of the camera sensor networks 610, 620, and 630 is able tocapture images of the unmanned aerial vehicle 104. Edge servers for thecamera sensor networks may transmit a notification to the cloud server102 that the unmanned aerial vehicle 104 is out of areas covered by thecamera sensor networks 610, 620, and 630. The cloud server 102 may theninstruct a mobile imaging device, e.g., an unmanned aerial vehicle 660to move towards the unmanned aerial vehicle 104. The unmanned aerialvehicle 660 includes an imaging device that captures images of theunmanned aerial vehicle 104. The unmanned aerial vehicle 660 maytransmit captured images to the user 160 via an edge server proximate tothe unmanned aerial vehicle 660 and/or the cloud server 102.

FIG. 7 is a flowchart for obtaining a view of an unmanned aerial vehicleusing camera sensor networks, according to another embodiment shown anddescribed herein.

In step 710, the system obtains an origin and a destination of anunmanned aerial vehicle. The unmanned aerial vehicle may be previouslyidentified by an edge server or imaging devices as described in step 510above. In embodiments, the unmanned aerial vehicle may communicate itsorigin and destination to an edge server or a cloud server. In someembodiments, the origin and destination of the unmanned aerial vehiclein association with the identification of the unmanned aerial vehiclemay be pre-stored in the cloud server 102.

In step 720, the system determines a group of imaging devices based on aroute between the origin and the destination of the unmanned aerialvehicle. In embodiments, the cloud server 102 may determine a route forthe unmanned aerial vehicle based on the origin and the destination ofthe unmanned aerial vehicle. For example, by referring to FIG. 6, thecloud server 102 may receive the origin 162 and the destination 164 ofthe unmanned aerial vehicle 104. The cloud server 102 may determine aroute including a path 602, a path 604, and a path 606. In embodiments,the cloud server 102 may determine imaging devices that are availablefor capturing images of the unmanned aerial vehicle 104 following theroute.

In step 730, the system determines whether one or more of the group ofimaging devices are unavailable. By referring to FIG. 6, the cloudserver 102 or an edge server for one of the camera sensor networks 610,620, 630 may determine whether one or more of the group of imagingdevices are unavailable.

If it is determined that the group of imaging devices are available atstep 730, the system obtains a view of the unmanned aerial vehiclefollowing the route based on images of the unmanned aerial vehiclecaptured by the group of imaging devices in step 740. For example, byreferring to FIG. 6, when the unmanned aerial vehicle 104 is at theorigin 162, the imaging device 610-1 initiates capturing images of theunmanned aerial vehicle 104. The imaging device 610-1 may transmit thecaptured images to an edge server for the camera sensor network 610 orthe cloud server 102 shown in FIG. 1 which in turn transmits thecaptured images to the device 163 of the user 160. As the unmannedaerial vehicle 104 passes the imaging device 610-1, the imaging device610-3 may capture images of the unmanned aerial vehicle 104 and transmitthe captured images to the edge server or the cloud server. As theunmanned aerial vehicle 104 follows the path 604, the imaging device620-1 may capture images of the unmanned aerial vehicle 104 and transmitthe captured images to the edge server or the cloud server. In thisregard, at least one imaging device in the camera sensor networks 610,622, 630 captures images of the unmanned aerial vehicle 104 in real timeand transmits the captured images to the user 160 such that the user 160can see the real time view of the unmanned aerial vehicle 104.

If it is determined that one or more of the group of imaging devices areunavailable at step 730, the system may select another group of imagingdevices in step 750. For example, by referring to FIG. 8, the imagingdevices 542 and 544 may not be available for capturing images of theunmanned aerial vehicle 104. For example, the imaging devices 542 and544 may not operate properly. As another example, the imaging devices542 and 544 may opt out from the camera sensor network 620. As anotherexample, the imaging devices 542 and 544 may be mobile imaging devicesand move out of areas covered by the camera sensor network 620 in FIG.6. The camera sensor network 620 in FIG. 6 may change to the camerasensor network 622 in FIG. 8 due to unavailability of the imagingdevices 542 and 544. In this case, the route of the unmanned aerialvehicle 104 may not be continuously monitored by the camera sensornetworks 610, 622, and 630 because of area not covered by any of thecamera sensor networks 610, 622, and 630. Then, the system may selectanother group of imaging devices, for example, imaging devices thatconstitute a camera sensor network 810 in FIG. 8.

In step 760, the system determines another route for the unmanned aerialvehicle based on locations of another group of imaging devices. Forexample, by referring to FIG. 8, based on the locations of imagingdevices for the camera sensor networks 810, a route including the path602, the path 604-1, a path 812, and the path 606-2 is selected for theunmanned aerial vehicle 104.

In step 770, the system transmits information about another route to theunmanned aerial vehicle. In embodiments, the cloud server 102 or an edgeserver proximate to the unmanned aerial vehicle 104 may transmitinformation about another route, e.g., the route including the path 602,the path 604-1, the path 812, and the path 606-2 to the unmanned aerialvehicle 104. Then, the imaging devices of the camera sensor networks610, 622, 810, and 630 may capture the unmanned aerial vehicle 104following the route.

FIG. 9 depicts a flowchart for obtaining a view of an unmanned aerialvehicle using camera sensor networks when the origin and destination ofthe unmanned aerial vehicle is not known to a system, according toanother embodiment shown and described herein.

In step 910, the system determines a route for the unmanned aerialvehicle based on at least one of a current location of the unmannedaerial vehicle, a trajectory of the unmanned aerial vehicle prior tocurrent location, and locations of available imaging devices. Inembodiments, the origin and destination of the unmanned aerial vehiclemay not be known to the system. By referring to FIG. 10A, the futureroute 1020 of the unmanned aerial vehicle may be predicted based on thetrajectory 1010 of the unmanned aerial vehicle prior to its currentlocation. For example, one or more imaging devices may monitor movementof the unmanned aerial vehicle 104. Based on the monitored movement ofthe unmanned aerial vehicle 104 including a moving direction, speed,etc., the edge server or cloud server may predict the route 1020 of theunmanned aerial vehicle 104 in real time.

In step 920, the system may select a group of imaging devices based onthe predicted route. For example, the cloud server 102 or the edgeserver proximate to the unmanned aerial vehicle 104 may select a groupof imaging devices proximate to the predicted route, e.g., imagingdevices 1030, 1032, 1034 in FIG. 10A.

In step 930, the system obtains a view of the unmanned aerial vehiclefollowing the route based on images captured by the group of imagingdevices. For example, by referring to FIG. 10A, the imaging device 1030initiates capturing images of the unmanned aerial vehicle 104 followingthe predicted route 1020. The imaging device 1030 may transmit thecaptured images to an edge server or the cloud server 102 shown in FIG.1 which in turn transmits the captured images to the device 163 of theuser 160. As the unmanned aerial vehicle 104 passes the imaging device1030, the imaging device 1032 may capture images of the unmanned aerialvehicle 104 and transmit the captured images to the edge server or thecloud server. The predicted route of the unmanned aerial vehicle 104 maychange in real time, and the system may select different group ofimaging devices based on the changed predicted route.

In some embodiments, the system may determine a route for the unmannedaerial vehicle based on locations of available imaging devices. Forexample, by referring to FIG. 10B, the origin and destination of theunmanned aerial vehicle is not known to the system. Then, the system maydetermine the locations of available imaging devices that are within acertain distance from the current location of the unmanned aerialvehicle 104, e.g., imaging devices 1040, 1042, and 1044. Based on thelocations of the imaging devices 1040, 1042, 1044, the system maydetermine the route 1050 for the unmanned aerial vehicle 104, andtransmit the route 1050 to the unmanned aerial vehicle 104. The imagingdevices 1040, 1042, and 1044 may capture images of the unmanned aerialvehicle 104 following the route 1050 and transmit captured images to theuser 160 via an edge server or the cloud server.

It should be understood that embodiments described herein are directedto methods and systems for obtaining a view of an unmanned aerialvehicle. The method includes obtaining an origin and a destination ofthe unmanned aerial vehicle, determining a group of imaging devicesbased on a route between the origin and the destination of the unmannedaerial vehicle, and obtaining a view of the unmanned aerial vehiclefollowing the route based on images of the unmanned aerial vehiclecaptured by the group of imaging devices.

The group of imaging devices form a collaborative camera sensor network.The collaborative camera sensor network identifies an unmanned aerialvehicle and creates a human operator's view of the unmanned aerialvehicle. The route of the unmanned aerial vehicle is determined based onthe availability of camera sensor networks. The route of the unmannedaerial vehicle may be changed based on the availability of imagingdevices. Whenever the unmanned aerial vehicle moves out the camerasensor networks region, horizontal and/or vertical hand over may beperformed between camera sensor networks as shown in FIG. 3 tocontinuously monitor the flight of the unmanned aerial vehicle. If theunmanned aerial vehicle is at a location not covered by the camerasensor networks, a special mission drone may be positioned proximate tothe unmanned aerial vehicle as shown in FIG. 4. The image data obtainedby the camera sensor networks is transferred to an edge server and/or acloud server which transmits the captured images to a human operator.

According the present disclosure, not only the route of the unmannedaerial vehicle but also the selected cameras can be changed upon theavailability of imaging devices. The imaging devices may be selectedbased on dynamic, static and/or predetermined metrics. Alternativeimaging devices may be selected when there is short-term and/orlong-term unavailability of participating imaging devices. According tothe present disclosure, the unmanned aerial vehicle is continuouslytracked and/or monitored via collaborative camera sensor networks andthe first-person view of the unmanned aerial vehicle is constructed fora human operator.

It is noted that the terms “substantially” and “about” may be utilizedherein to represent the inherent degree of uncertainty that may beattributed to any quantitative comparison, value, measurement, or otherrepresentation. These terms are also utilized herein to represent thedegree by which a quantitative representation may vary from a statedreference without resulting in a change in the basic function of thesubject matter at issue.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

What is claimed is:
 1. A method for generating a view for an unmannedaerial vehicle, the method comprising: obtaining an origin and adestination of the unmanned aerial vehicle; determining a group ofimaging devices based on a route between the origin and the destinationof the unmanned aerial vehicle; and obtaining a view of the unmannedaerial vehicle following the route based on images of the unmannedaerial vehicle captured by the group of imaging devices.
 2. The methodof claim 1, wherein the group of imaging devices is determined based onat least one of locations of the group of imaging devices, predictedlocations of the group of imaging devices, or specifications of thegroup of imaging devices.
 3. The method of claim 1, wherein thespecifications include at least one of an angle of view, resolution, ora viewing distance.
 4. The method of claim 1, wherein: the group ofimaging devices include one or more mobile imaging devices, and the oneor more mobile imaging devices are expected to be proximate to the routewhen the unmanned aerial vehicle travels along the route.
 5. The methodof claim 4, wherein the one or more mobile imaging devices include atleast one of an imaging device of a mobile phone, an imaging device ofanother unmanned aerial vehicle, or an imaging device of a vehicle. 6.The method of claim 1, wherein obtaining the origin and the destinationof the unmanned aerial vehicle comprises: identifying the unmannedaerial vehicle; and retrieving the origin and the destination of theidentified unmanned aerial vehicle.
 7. The method of claim 1, furthercomprising: selecting another group of imaging devices in response tounavailability of one or more of the group of imaging devices proximateto the route; determining another route based on locations of theanother group of imaging devices; and transmitting information about theanother route to the unmanned aerial vehicle.
 8. The method of claim 1,wherein the group of imaging devices includes at least one of a imagingdevice at a traffic light, an imaging device of a security camera, animaging device of a roadside camera, an imaging device of a mobilephone, an imaging device of another unmanned aerial vehicle, or animaging device of a vehicle.
 9. The method of claim 1, furthercomprising: comparing images of the unmanned aerial vehicle captured bythe group of imaging devices; identifying a misbehaving imaging deviceof the group of imaging devices that provides images not congruent withimages provided by other imaging devices of the group of imagingdevices; and discarding the images provided by the misbehaving imagingdevice.
 10. The method of claim 1, wherein determining the group ofimaging devices based on the route between the origin and thedestination of the unmanned aerial vehicle comprises: determininginformation about availability of imaging devices based at least one oflocations, predicted locations, and face directions of the imagingdevices; and selecting the group of imaging devices based on theinformation about availability of imaging devices.
 11. The method ofclaim 10, further comprising: providing incentives to the selected groupof imaging devices.
 12. The method of claim 11, wherein the incentivesare determined based on at least one of a distance between the route andeach imaging device, a view angle of each imaging device, or a viewingdistance of each imaging device.
 13. The method of claim 1, furthercomprising: determining whether the unmanned aerial vehicle is within aview of at least one of the group of imaging devices; and dispatchinganother unmanned aerial vehicle proximate to the unmanned aerial vehiclein response to the unmanned aerial vehicle being outside views of thegroup of imaging devices.
 14. A system for generating a view for anunmanned aerial vehicle, the system comprising: an electronic controlunit configured to: obtain an origin and a destination of the unmannedaerial vehicle; determine a group of imaging devices based on a routebetween the origin and the destination of the unmanned aerial vehicle;receive images captured by the group of imaging devices; and obtain aview of the unmanned aerial vehicle following the route based on thereceived images.
 15. The system of claim 14, wherein the electroniccontrol unit is further configured to: select another group of imagingdevices in response to unavailability of one or more of the group ofimaging devices proximate to the route; determine another route based onlocations of the another group of imaging devices; and transmitinformation about the another route to the unmanned aerial vehicle. 16.The system of claim 14, wherein the electronic control unit is furtherconfigured to: instruct the group of imaging devices to store the imagesfor a predefined amount of time.
 17. The system of claim 14, wherein theelectronic control unit is further configured to: determine whether theunmanned aerial vehicle is within a view of at least one of the group ofimaging devices; and transmit an alert in response to the unmannedaerial vehicle being outside views of the group of imaging devices. 18.A method for generating a view for an unmanned aerial vehicle, themethod comprising: determining a route for the unmanned aerial vehiclebased on a current location of the unmanned aerial vehicle; selecting agroup of imaging devices based on the route; and obtaining a view of theunmanned aerial vehicle following the route based on images captured bythe group of imaging devices.
 19. The method of claim 18, furthercomprising: determining the route for the unmanned aerial vehiclefurther based on a trajectory of the unmanned aerial vehicle prior tothe current location.
 20. The method of claim 18, further comprising:determining the route for the unmanned aerial vehicle further based onlocations of available imaging devices.